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ABSTRACT 

This document presents the written records of the Third 
Annual National Institute for Science Education (NISE) Forum on indicators of 
success in postsecondary science, mathematics, engineering, and technology 
(SMET) education. The primary goal of this Forum was to initiate a national 
dialogue about how assessment and evaluation are and should be used to foster 
improvements in SMET education at all levels in the U.S. higher education 
systems. The records, all of which appear in this Proceedings, comprise the 
opening keynote, a digest of the three panel discussions, the remarks of 
panel discussants, a synthesis of participants 1 observations written after 
each panel discussion (think pieces), the closing reflections, an analysis of 
the participants' theories of change, and the presenters’ papers. A list of 
acronyms and a list of the approximately 300 participants’ names and location 
information are provided in the appendix. The panelists papers include: (1) 

’’Assessment as a Learning Process: What Evidence Will We Accept That Students 
Have Learned?’’ (Diane Ebert-May) ; (2) ’’Moving the Mountain: Impediments to 

Change’’ (Eric Mazur); (3) "The Integrality of Assessment” (David B. Porter); 
(4) "The Reaction to the Symptoms Versus the Reaction to the Disease" (Brian 
P. Coppola); (5) "Assessment and the Promotion of Change in Community 
Colleges" (Eileen L. Lewis); (6) "Assessing and Evaluating the Evaluation 
Tool: The Standardized Test" (Richard Tapia); (7) "Assessment, Evaluation and 
Accreditation: Are We on the Same Wave Length? Or How Does One Provide 
Linkages for Systemic Change?" (Jack Bristol); (8) "An Assessment Model to 
Drive Undergraduate Educational Reform in the SMET Fields in a Large Public 
Multicampus University System" (Manuel Gomez); and (9) "Technology-Assisted 
Learning in Higher Education: It Requires New Thinking about 
Assessment" (Sheri D. Sheppard) . (MM) 
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Introduction: A Framework for Forum Contributions 



Elaine Seymour 

This document presents the written 
records of the Third Annual National 
Institute for Science Education (NISE) 
Forum on indicators of success in 
postsecondary science, mathematics, 
engineering and technology (SMET) 
education. The records, all of which appear 
in this Proceedings, comprise the opening 
keynote, a digest of the three panel 
discussions, the remarks of panel 
discussants, a synthesis of participants’ 
observations written after each panel 
discussion (think pieces), the closing 
reflections, an analysis of the participants’ 
theories of change, and the presenters’ 
papers. A list of acronyms and a list of the 
approximately 300 participants’ names and 
location information are provided in the 
appendix. 

Arthur Ellis, Elaine Seymour, and 
Susan Millar led the Forum development 
team. This core group was assisted by 
program officers from the National Science 
Foundation’s (NSF) Division of 
Undergraduate Education and by the 
following members of NISE’s College Level 
One team: Aaron Brower, Ann Burgess, 
Anthony Jacob, Kate Loftus, Robert 
Mathieu, and Catherine Middlecamp. 

Forum Goal and Design 

The primary goal of this Forum was to 
initiate a national dialogue about how 
assessment and evaluation are and should 
be used to foster improvements in SMET 
education at all levels in the U.S. higher 
education system. To focus attention on 
different levels within the system, three 
different but related topics — assessment at 
the classroom level; assessing learning as 
an aspect of change in classrooms, 
disciplines, and institutions; and the role of 
evaluation in change at the department, 



institutional, and national levels — were 
chosen. The following strategies were used 
to foster productive and genuine dialogue 
about these topics: 

• Panelists and panel facilitators were 
selected on the basis of their rich 
experiences with assessment and 
evaluation and their diverse 
backgrounds within higher education. 
The panel facilitators were Brock 
Spencer, professor of chemistry and 
principal investigator of the NSF- 
funded ChemLinks Chemistry Systemic 
Reform project, Beloit College; Elaine 
Seymour, director of the Ethnography 
and Evaluation Research Center at the 
University of Colorado at Boulder; and 
Clifford Adelman, Senior Research 
Analyst at the Office of Educational 
Research and Improvement, U.S. 
Department of Education. 

• Scientists, mathematicians, engineers 
and administrators from many 
disciplines were invited by contacting 
members of higher education 
organizations such as the American 
Association of Higher Education 
(AAHE), the American Geological 
Institute, the American Chemical 
Society, and the American Physical 
Society. 

• Panelists were asked to write short 
papers for participants and the other 
panelists to read ahead of time. 

• The facilitators worked with the 
panelists to plan a “fishbowl” format 
that allowed them to explore a set of 
linked themes developed in their 
various papers, and also to engage the 
audience in the discussion. 

• Officers of the NSF were asked to frame 
the discussion, with the agency’s 
director, Neal Lane, and its Assistant 
Director for Education and Human 
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Resources, Luther Williams, opening 
the Forum with a plenary address, and 
with Norman Fortenberry (Director, 
Division of Undergraduate Education), 
Daryl Chubin (Director, Division of 
Research, Evaluation, and 
Communication), and Larry Suter, 
(Deputy Director, Division of Research, 
Evaluation and Communication) 

• Two institutional administrators, Cora 
Marrett, Provost, University of 
Massachusetts-Amherst, and John 
Wiley, Provost, University of Wisconsin- 
Madison, were chosen to provide a real- 
time syn thesis and closing remarks. 

• Participants were assigned to 
discussion groups that met after each 
panel session. They were asked to spend 
the first fifteen minutes responding to 
the panel in written think pieces that 
were used to produce the Synthesis 
portion of this document. The 
remainder of each discussion group 
session was spent sharing knowledge 
and views. 

• Participants were invited to present 
poster sessions for perusal by others 
during a lengthy lunch period. 

A second goal of this Forum was to 
inform the NISE’s College Level One team 
of the needs of the postsecondary SMET 
education community with respect to 
assessment. This information is being used 
by the College Level One team’s 1998-99 
Institute on Assessment in College Level 
SMET Classrooms. For general information 
about this team and its work following on 
this February 1998 Forum, readers may 
wish to consult the NISE website 
(, http://www.wcer.wisc.edu/nise ), and the 
College Level One team’s website 
(http://www. wcer. wise. edu/nise/CLl /). 

Guide to the Reader 

This document is intended primarily to 
provide the participants, officers of the 



NSF, members of the NISE, and the 
national community with a thematic 
account of the ideas and issues developed 
for and during this Forum. Each section of 
this document considers, from a different 
perspective, the major issues raised, 
courses of action undertaken or advocated, 
areas of agreement and dissent, 
expectations, and unmet needs. A few 
specific comments about the Panel 
Discussion Summaries and Commentaries, 
the Synthesis, and the Analysis of 
Participants’ Theories of Change sections 
are provided to guide the reader. 

The Panel Discussion and Summaries 
section is designed for readers seeking an 
understanding of the key points made 
during the panel sessions. Each of the three 
summaries presents the key themes and 
points made by the panelists. The 
Commentary following each summary is the 
full transcript of the remarks made by the 
panel discussant. As the panelists’ remarks 
were made on the assumption that the 
participants had read the panelists' papers, 
readers may find it useful to read the 
relevant papers (see Appendix A) in 
conjunction with the summary of each 
panel. 

The Synthesis of Participants’ Think 
Piece Essays section is designed for readers 
with a special interest in understanding the 
views of the assembled 300 forum 
participants. The majority of Forum 
participants were faculty and 
administrators from four-year colleges and 
universities. A number of representatives 
from two-year colleges, K-12 educational 
systems, national agencies, and 
professional organizations were also in 
attendance. This section synthesizes the 
salient points appearing in the nearly 600 
different think pieces written by the 
participants after the three panel sessions, 
attempting to include points made by 
participants from each of these stakeholder 
groups. As the think piece essays were 
written in response to, and frequently 
referred to specific points made in, the 
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panel discussions, those who read both the 
panel summaries and the synthesis will 
note that forum participants reinforced 
many of the same points made by the 
panelists, but also raised important issues 
not addressed by the panelists. 

The Analysis of Participant Theories of 
Change offers a broad conceptual 
framework for the key points made in all 
the components appearing in this 
Proceedings, and integrates themes 
developed by all Forum participants. 
Readers with an interest in the process of 
higher education reform will find this piece 
of special interest. 

Main Themes and Voices from the 
Forum 

The Centrality of Assessment in Higher 
Education Reform 

That a forum focused on assessment 
should be convened at this time reflects a 
particular stage in the nationwide 
conversation among postsecondary 
educators about the current and future 
expectations and needs of higher education 
and the role of SMET education within it. 
Assessment is an appropriate focus for a 
discussion of next steps because, as our 
collective experience with reform issues has 
grown, we have come to understand that 
issues of measurement and evaluation are, 
as panelist Brian Coppola observed, 
actually or potentially “linked to every 
other aspect of the instructional setting.” 
The Forum participants identified the 
following functions of assessment as critical 
in the endeavor to enhance the quality of 
SMET higher education: 

• exploration and clarification of student 
learning gains in conceptual 
understanding and in the ability to 
make connections, formulate research 
questions, and communicate knowledge; 
and 
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• evaluation of teaching in ways that 
more accurately reflect its efficacy in 
enabling learning. 

Many participants identified as new to 
the higher education SMET culture a 
growing realization that these two types of 
assessment — of students and of faculty — 
should focus on learning and are essentially 
linked. Participants proposed that 
assessment data from these two sources 
should be used: 

• to provide useful feedback to both 
teachers and students on their work; 

• to establish the worth of reformed 
teaching activities; 

• as a basis for argument and persuasion 
to promote further improvements in the 
quality of SMET education; 

• to protect from negative career 
consequences faculty who work in 
contexts that are unsympathetic to their 
classroom innovations; and 

• as primary resources for institutions, 
disciplines, and national agencies that 
need data gathered at classroom, 
departmental, and institutional levels 
to develop aggregate measures of 
progress. 

The Search for Indicators at All System 
Levels 

Forum participants were hopeful that 
assessment could provide coherent, workable, 
and cost-effective indices of progress and 
accountability for individual faculty, 
departments, and whole institutions. Such 
indices must be grounded in more accurate and 
■' transferable measurement of student learning 
gains. They also must be capable of assessing 
the effectiveness with which the system is 
“making provision” for SMET education that is 
of high quality yet affordable (Luther Williams, 
opening keynote). 

The Forum participants agreed that the 
process of developing cost-effective, accurate, 
and transferable indicators of system quality is 
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challenging. Larry Suter (discussant) observed 
that he has been actively addressing this 
challenge since 1990, when he began turning 
more attention to the national data about the 
undergraduate sector. He found that he “could 
not report how much our students were 
learning, how the pipeline was working, or 
where the holes were. . . . There were lots of 
statistics. . . . What was missing was the 
organization of information around subject 
areas.” Clifford Adelman (panel facilitator) 
agreed that “without some metrics for 
aggregation, all you have are fragments.” In 
addressing the challenge to improve these 
data, Suter said that the NSF has experienced 
difficulty finding people “who are willing to 
tackle the measurement problem” facing 
postsecondary education. He observed that in 
the development of measurements, there is not 
yet “the same kind of leadership in the higher 
education area as there is in elementary and 
secondary education.” However, Suter was 
hopeful that a number of recently funded 
projects would develop national indicators that 
give a better public account of higher education 
teaching and learning. In addition, he hoped 
that the Forum would encourage more people 
to become engaged in discussions about 
measurement. 

In Williams’ view, “the two domains” of 
student learning gains and system indicators 
“interconnect, but one does not substitute for 
the other.” Other contributors were hopeful 
that common measures of student learning 
used across departments could collectively 
provide systemic indicators of what Manuel 
G6mez (panelist) called the institution’s 
educational health. The ongoing task for 
education scholars, assessment and evaluation 
specialists, funding agencies, and the reform 
community is to grapple with the difficulties of 
how to develop measurements that will serve 
every level of the system and the system 
overall. 



Cross-Currents: The Viewpoints, 
Expectations, and Needs of Participants 

The papers, think pieces, panel discussions, 
and observations from the floor indicated 
several subsets of interest in assessment and 
differences in the nature of the participants’ 
engagement with assessment and with 
curriculum reform issues. The viewpoints 
outlined below reflect the wide invitational 
character of the Forum, and differences in the 
nature of the participants’ engagement with 
issues of educational quality and change. 

Deeply Engaged Voices 

Some views reflected deep and long 
engagement with the improvement of SMET 
higher education and with the standards by 
which it is judged. These observers took it as a 
given that changes were underway. Many were 
highly engaged in reform activities as teachers 
researchers, and administrators. Some looked 
to assessment as the means to inform, 
improve, and validate their own work and to 
move the reform movement forward. Some 
expressed regret that the Forum did not 
address in a more systematic way what is 
already known about assessment and 
evaluation methods applied specifically within 
higher education. They strongly debated 
various strategies for classroom-level 
assessment and for leveraging change at the 
departmental and institutional levels. 

Newly Engaged Voices 

The Forum also included faculty who had 
more recently discovered what panelist Diane 
Ebert-May and others referred to as “active 
learning.” Their expressed expectation was 
that the Forum would offer practical guidance 
in how to better design and use assessments. 
While they gained knowledge of tools and 
instruments that they could adapt for their 
own classroom use from the panel and poster 
sessions, they also looked for more. They used 
the small-group discussions and informal 
encounters as opportunities for networking am 
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exchange of assessment materials and 
methods. This clearly expressed need speaks to 
the recommendations made by a number of 
speakers that national and institutional 
leadership should increase access to workshops 
on curriculum development, pedagogical 
techniques, and the design of learning 
assessments. 

Gradualist Voices 

A third set of participants was less 
concerned with “change” or “reform” than with 
assessment in the service of enhanced cur 

riculum quality. As John Wiley (one of the 
closing speakers) emphasized, better 
assessment techniques would allow clearer 
judgement of the relative merits of different 
approaches to teaching — whether more or less 
traditional. These gradualist voices also 
reminded the Forum participants that large 
and important sections of the SMET 
community remain skeptical about the need for 
curriculum reform or for a new emphasis on 
learning, and that a temperate choice of 
language in advocating these ideas may be 
wise. As William Clune (participant) wrote, 
“We’re talking about careful change.” 

Community College and K-12 Voices 

Participants whose comments reflect the 
experience of SMET teachers in community 
colleges and the K-12 system urged reformers 
to take into account the needs of their students 
and to learn from K-12 teachers and 
community college instructors, who have much 
longer experience with curriculum reform and 
professional development. They pressed for the 
alignment of the new goals and practices in 
four-year higher education institutions with 
those of the two-year college systems. Eileen 
Lewis (panelist) pointed out that the prevailing 
alignment may not be in the desired direction. 
The prestige of research institutions is such 
that their pedagogical strategies exert a 
conservative influence on teaching methods in 
community colleges. 



Voices Calling for the Evaluation of Evaluation 
Criteria 

The last voice, heard from across all 
groups, expressed dissatisfaction with the 
following common assessment practices: 

• Use of standardized tests (such as the SAT) 
as an “inappropriate filter” that effectively 
deprives institutions of talented students 
who have poorer access to good college 
preparation. Panelist Richard Tapia argued 
that the effect of this practice is to reinforce 
class and race/ethnic bias in access to 
quality higher education. 

• Use of institutional assessment 
instruments whose purpose is obscure and 
methods are faulty. Participants were 
concerned that the use of poorly designed 
instruments has left faculty indifferent or 
suspicious of assessment in general and, 
thus, disinclined to consider the use of 
institutional assessment practices, 
regardless of their quality. 

• Use of poorly designed end-of-semester 
course evaluation instruments. Faculty 
reported that these instruments commonly 
fail to give information about student 
learning gains, offer poor feedback (except 
in the students’ write-in comments), and 
sometimes are used inappropriately in 
tenure or promotion decisions. 

All of these commentators’ objections to 
institutional assessment focused on the poor 
quality of many of the instruments they are 
obliged to use. Participants called for re- 
examination of the criteria on which commonly 
used assessment instruments are based and 
evaluation of the consequences of using these 
instruments. As Tapia argued, “The criteria 
they reflect can hinder outcomes that we 
value.” 

The reader will find these themes and 
these voices recurring throughout the 
document that follows. Together, they 
reflect the stage that we have reached in 
our common search for improved quality, 
and for indicators thereof, in SMET higher 
education. 
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I am pleased to join you again. I had the 
opportunity to make comments at the first 
two Forums and certainly look forward to 
this one also being a very productive event. 
Even though the Institute is only in its third 
year, it is very important that, after having 
given the attention initially to the K-12 
sector, the Institute moves to extend 
attention so that it is actually concerned with 
the entire continuum. Without any debate, 
many of the issues that you labeled K-12 in 
your very excellent summary are not K— 12; 
they are at least K-16. I also applaud you for 
focusing on a very manageable agenda. In an 
arena as diverse and complex as 
undergraduate science, mathematics, 
engineering, and technology education, you 
did not concentrate on the areas in which 
there is an abundance of knowledge. Rather 
you engaged a topic about which I would 
submit we know very little: success in 
general, particularly the indicators of success 
that have been tested in any substantive 
fashion. What we know in a tested fashion in 
this arena is almost zero. 

I have three points to make. First, the 
work that you are going to take up here, 
focusing on the undergraduate sector, maps 
very well with a report that the Directorate 
for Education and Human Resources (EHR) 
commissioned, Shaping the Future: Strategies 
for Revitalizing Undergraduate Education 
(NSF 98-73). The work was done by the 
Advisory Committee to the Directorate, led 
by Mel George. Several years ago we asked 
this very eclectic group of scientists, 
engineers, mathematicians, and educators to 
consider the needs of all undergraduates 
attending all types of U.S. two- and four-year 
colleges and universities. To do that, 
obviously, the study had to be concerned with 
the needs of majors in science and 



engineering, with the preparation of K-12 
mathematics and science teachers and their 
continued professional development, with the 
needs of the individuals who enter the 
technical workforce, and with the issue of 
science literacy in general, 

The recommendations of the Advisory 
Committee were that all students have 
access to supportive, excellent Science, 
Mathematics, Engineering, and Technology 
(SMET) undergraduate education and, to the 
extent possible, that all students learn these 
subjects with direct experience with the 
methods and the processes of inquiry. Both 
recommendations are very challenging. The 
second one, in particular, requires the 
aggregate output of the expertise represented 
by the people gathered here. Certainly, we 
have not yet established supportive, excellent 
SMET undergraduate education for all 
students, and occasionally that education 
provides little positive attention to processes 
and methods of inquiry. 

The Advisory Committee made a series of 
recommendations addressed to a variety of 
sectors, but some of those recommendations 
were to NSF, and several of those are 
relevant to this conference. It was 
recommended that NSF, employing all of its 
resources, especially those in the research 
directorates, increase research on the 
undergraduate educational experience in 
disciplinary and interdisciplinary contexts, 
focusing on efforts to integrate research and 
education. More generically, it recommended 
that NSF provide support for a research 
agenda on human learning at the 
undergraduate level; use the results to 
evaluate programs, both long term and short 
term; and develop indicators of success to 
guide future program development. 



My second point is that the agency has 
attempted to respond to most of those 
recommendations. We have placed increased 
emphasis on integrating research and 
education. Drawing on the resources of EHR, 
joined with the efforts of the research 
directorates, the research agenda is being 
pursued in a variety of arenas. One of the 
other recommendations has to do with 
supportive and high quality education for all 
students. An example far removed from 
Education and Human Resources (EHR), but 
extremely important, is the recent decision 
by the Engineering Directorate to make the 
transition from the Engineering Coalitions, 
which you are familiar with, to the daunting 
effort of systemic reform of undergraduate 
engineering education. Of course, that’s an 
untenable agenda pursued by engineering 
alone, because one has to give systemic 
attention to all of science and mathematics to 
accomplish reform of engineering. 

In EHR and the rest of the directorates, 
the broad interdisciplinary theme of 
Knowledge and Distributed Intelligence 
seeks to integrate the knowledge base for 
issues from the neurosciences — the human 
brain, its functions, its patterns of 
expression — to the important research 
bearing on cognitive processes, to how one 
employs learning technologies in artificial as 
well as human settings, and finally to how 
students learn in different physical settings, 
cultural settings, and so on. Integral to that 
research, it seems to me, is to couple it with 
what it portends in terms of outcomes. Stated 
differently, how does one, with the enhanced 
knowledge base, reconfigure the programs, 
and then, how does one assess their value in 
terms of student learning? 

We are very concerned with 
disseminating the excellent products that you 
are responsible for, and we have used rather 
traditional methods to do so. But presently 
we are devoting attention, in collaboration 
with the Computer and Information Science 
and Engineering (CISE) Directorate, to 
actually bring the dissemination agenda to 
what we now call a SMET Digital Library, 



kindergarten through probably graduate 
education. The principal programs in the 
undergraduate component of NSF — faculty 
development, course and curriculum 
development, and instrumentation and 
laboratory improvement — are increasingly 
being configured in a continuum. In other 
words, we are trying to find ways to enham 
the coherence between faculty development 
needs and course and curriculum 
development to optimize student learning, i 
a part of the process needed to implement t 
recommendations of Shaping the Future is 
under way. 

We also need continued advice from yoc 
in a variety of other sectors. Last fiscal yea 
we introduced a process of program 
effectiveness reviews to serve as a precursc 
to the Government Performance and Resul 
Act, a congressionally mandated process b> 
which agencies defend their budgets in an 
outcomes context. We have conducted a 
variety of these reviews. I want to refereno 
two of them that we held recently. One was 
an examination of the course and curriculu 
portfolio in the Division of Undergraduate 
Education (DUE). Another was an 
examination of the research program withi 
the EHR Directorate. In combining the 
results of these two reviews we concluded 
that we need to find robust mechanisms to 
increasingly advance the knowledge base 
that undergirds the investment we make. r 
say it another way, while the Institute’s w< 
is exceedingly important, there is somethir 
less than an appropriate representation of 
research devoted to practice in the 
undergraduate sector. (I do not imply that 
there’s an abundance of such research evei 
in the K-12 sector.) 

My third point is, What do points one a 
two suggest for your agenda? I’ve already 
indicated that your focus on indicators of 
success is important and that we know vei 
little about indicators. A problem-solving 
design would justify starting with indicato 
You have to acknowledge not inputs but 
student learning. What would be definitive 
effective, durable measures of success? Thi 
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indicators have to be organized into broad 
rubrics in two domains. One certainly has to 
do with student performance as a measure, 
but another one has to be system based. It 
has to assess the effectiveness of the system 
responsible for making provisions for quality, 
supportable SMET education. The two 
domains interconnect, but one obviously 
doesn’t substitute for the other. In my 
judgment, that’s a major challenge. 

My purpose in raising this third point is 
to bring to your deliberations some emphasis 
on system thinking. Assessment framed in 
terms of outputs, that is student learning, 
does not frame improving undergraduate 
education. The issue is a question of 
specificity. Obviously individual institutions, 
programs, and departments need strategies 
and mechanisms to report their performance. 
These reports need to be aggregated in some 
fashion by which we can assess the SMET 
enterprise in the nation. Here again I’m 
emphasizing that student learning and 
improved undergraduate education do not 
have the same set of indicators. Whatever 
mechanisms are used will have to be • 
thoughtfully developed, so that we don’t limit 
the possibilities or constrain the design. 

What would be reasonable indicators of 
success from a student learning perspective 
at the undergraduate level? Which 
students — majors, mathematics/science 
teachers, generic matriculants, those seeking 



science literacy in general? Undergraduate 
education is a very diverse enterprise. What 
is the undergraduate analog to the SAT? 
What would be effective indicators? We have 
to be very creative, innovative, and in large 
measure unconstrained by the self-evident to 
arrive at indicators that actually are going to 
be useful. And those indicators should not be 
independent of the fact that the SMET 
undergraduate enterprise is dynamic. 
Matching the indicators to an enterprise that 
does not represent a fixed set is challenging. 

You have a variety of very good examples 
of the leaders of the change represented in 
the Forum. I urge you to combine 
considerations of student learning and of 
improved undergraduate education so that 
you do not develop indicators for the 
enterprise of years past. If it is taken up, the 
combination would allow you to link your 
findings to the research that has already 
been done on the K-12 sector. It would be 
consistent with what I anticipate will be 
some of the major issues that will be 
addressed in the science and engineering 
Graduate Education Forum coming up in 
June. 

I thank you for your continued 
engagement in this arena, making very clear 
NSF’s continued interest and support. I look 
forward to the products of your work. Thank 
you very much. 
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Panel Discussion Summaries and Commentaries 

Panel 1: Assessment of Teaching, Learning, 
and Curriculum Change in SMET Classrooms 1 



Panel Discussion Summary 

Susan B. Millar 



Panel Facilitator: Brock Spencer, Kohnstamm Professor of Chemistry and 

Associate Dean of the College, Beloit College 
Panel Members: Diane Ebert-May, Director and Associate Professor, 

Science & Mathematics Learning Center and Department 
of Biological Sciences, Northern Arizona University 
David B. Porter, Professor, Department of Behavior Sciences 
and Leadership, U.S. Air Force Academy 
Eric Mazur, Gordon McKay Professor of Allied Physics 
and Professor of Physics, Harvard University 



Abstract 

Panelists used a broad definition of 
course-level assessment, suggesting that 
assessment practices need to be woven 
through the fabric of the learning process 
rather than be conceived as involving only 
graded evaluation activities. They explained 
that assessment practices need to be aligned 
with faculty and institutional goals for 
student learning, and that good assessment 
methods help both learners and teachers 
become more reflective practitioners. They 
outlined three steps: decide on course goals, 
communicate these goals to students through 
both word and example, and use teaching 
strategies designed to achieve the goals. The 
panelists favored more invitational classroom 
assessment practices that challenge students 
to develop higher-level thinking skills without 
inducing destructive competition among 
students. They emphasized the importance of 



fostering student and faculty buy-in and 
acknowledged some of the risks posed when 
faculty adopt new methods. They suggested 
that faculty interested in using these new 
methods build supportive networks by linking 
with like-minded faculty and administrators 
to achieve a critical mass and by being 
prepared for the ups and downs inherent in 
the change process. Finally, they noted that 
course-level changes in postsecondary Science, 
Mathematics, Engineering, and Technology 
(SMET) education need to be aligned with K- 
12 SMET reform more generally. 

Spencer began the discussion by inquiring 
how well traditional assessment methods 
inform faculty of what students are learning. 
The panelists responded by considering 
multiple-choice exams, stating that this 
assessment method fails to provide 
instructors with either (1) adequate 
information about higher-level abilities, such 



1 Although the title of this session suggests a focus on assessment indicators, the panel discussion actually focused on 
how to implement reform at the course level. The papers written by the three members of this panel appear in Appendix 
A and provide classroom-based examples of assessment practices. 
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as critical thinking skills, or (2) substantive 
feedback about student learning and their 
teaching. Moreover, the panelists asserted 
that multiple-choice exams and similar 
assessment methods create a “conspiracy of 
mediocrity” that fosters superficial learning. 
Such traditional assessment methods imply 
that the instructor’s task is to package and 
transfer information to students who should 
dutifully receive, retain, and reproduce it in 
the final exam. 

Need for Alignment 

Having agreed among themselves that 
traditional forms of assessment generally are 
inadequate for current educational goals, the 
panelists presented strategies for improving 
assessment activities. They agreed that the 
essential first step was for faculty to 
articulate their learning goals before 
considering assessment activities. They 
stressed that these goals should be aligned 
with both student needs and institutional 
goals. Once these goals have been 
articulated, faculty can design assessment 
strategies that are “aligned” with them. This 
requires, as Ebert-May explained, that 
“learning outcomes be clearly defined in 
operational terms by specifying what criteria 
for performance you will accept at which 
points in the course to award a C, and what 
criteria for an A.” 

The panelists emphasized that, as faculty 
articulate new learning goals and develop 
assessment strategies in alignment with 
these goals, they must also use teaching 
strategies that serve these ends. Each 
panelist discussed from personal experience 
teaching strategies that are commensurate 
with particular learning goals. 

One strategy that Ebert-May has found 
productive is to communicate to the students 
both the goals of the course and the 
assessment criteria — in the form of a scoring 
rubric — by which they will be evaluated. In 
so doing, instructors give students road maps 
for the paths they are about to travel and 



share with them the responsibility for 
achieving the course goals. In her view, it is 
especially important for faculty who are 
experimenting with new ways of teaching 
and learning to use this strategy. Extending 
Ebert-May’s point, Mazur argued that, as 
well as sharing their evaluation criteria with 
students, teachers should offer them 
examples of excellent responses to homework 
and exam problems and other types of work 
expected of them. 

Mazur described a teaching method he 
uses to achieve his goal of helping students 
develop the habit of seeking out information, 
rather than merely receiving it. To help 
students learn to synthesize information 
from resources, he requires them to develop i 
two-paragraph summary of the reading 
before lecture. On a spot-check basis, 
students earn bonus points for their 
summaries. 

David Porter advocated the use of 
teaching strategies that “walk our talk.” He 
argued that “the scientific method is more 
likely to be caught than to be taught — our 
students learn a lot from what we do.” 

The heart of the scientific method is how we 
view mistakes: we honestly endeavor to find 
evidence contrary to our assumptions. We 
need to do that as faculty when we work wit! 
students in the classroom — and also as 
reformers, when we work with fellow faculty. 
We need to create a climate where both our 
own classroom and lab mistakes, and those o 
our students, are appreciated for the 
opportunities for learning that they bring. 
Viewing our work as teachers in this way 
involves a paradigm shift. Often we assume 
that we, the learned, are there to give 
something to the unlearned, the students. 
That assumption is in itself a major obstacle 
real education taking place in the classroom 

The panelists also described how they 
design their assessment strategies so as to 
mitigate student resistance to new teachinj 
and learning strategies. Each noted that th 
students most likely to resist new methods 
are those who learned how to perform at 
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especially high levels in traditional SMET 
courses and those who are taking SMET 
courses as prerequisites foricompetitive 
majors. Each panelist offered different 
assessment strategies. 

Porter explained that his assessment 
strategies are designed to (1) communicate 
clearly to the students what he values, (2) 
answer the question, “How is each student 
good?” by using diagnostic tools that also are 
intended to engage students in learning, and 
(3) answer the question, “How good is each 
student?” by using evaluative tools that 
inevitably are also anxiety-producing. Thus, 
his assessment activities convey, encourage, 
and reward the new kind of learning he seeks 
to achieve. 

Mazur starts his course with a 
questionnaire that asks students, in part, to 
describe their course goals. In the next 
lecture he presents an analysis of their goal 
statements (contrasted with his own goals) 
and suggests that the course will work better 
if the students’ and instructor’s goals match. 
A month later, he administers a second 
questionnaire that includes the question, “If 
you were teaching this course, what would 
you do?” Typically, he learns that most 
students say they would do what he is doing 
and only describe a few things that bother 
them. He then addresses these problems in 
the next lecture. (Details on this method are 
presented in Mazur’s book Peer Instruction: A 
User’s Manual.) 

As already noted, Ebert-May spends 
significant time at the beginning of a course 
explaining her course goals. Throughout the 
course she requires the students to engage in 
self-reflection about the degree to which they 
think they are accomplishing these goals. 
Self-reflection encourages students to take 
responsibility for understanding what they 
need to do and how they need to proceed in 
order to accomplish the goals. She finds that 
even resistant preprofessional majors begin 
to function as reflective learners: “Their 
attitudes change 180 degrees,” constituting a 
paradigm shift not only for the students, but 
also for the faculty members whose reform 



efforts are strongly reinforced by positive 
changes in their students. 

With respect to the idea of fostering 
reflection as a learning strategy, Clifford 
Adelman, a participant, described how using 
an experimental examination called 
“Academic Competencies in General 
Education” fostered faculty reflection. 2 A 
faculty panel developed assessment questions 
by means of sorting, by cognitive complexity, 
large numbers of student answers to open- 
ended questions. This process led the faculty 
panelists to become more reflective about 
how students actually think and to realize 
the need to help students recognize naive 
assumptions and use the scientific method to 
arrive at verifiable assumptions. 
Subsequently, in a half-dozen workshops, 
faculty from some 70 institutions used the 
process, principally for faculty development. 
Adelman commented that the purpose of 
assessment at the classroom level is to 
encourage both students and faculty to 
engage in analysis processes that help them 
understand the students’ current and 
evolving assumptions and knowledge. This 
purpose contrasts with the more traditional 
purpose of getting all the students to 
demonstrate that they can perform the same 
set of predetermined tasks. 

Ebert-May concurred with Adelman: she 
has successfully used a similar sorting 
process with departmental peers to develop a 
shared and grounded understanding of what 
it means to give a grade of C, B, or A. She is 
finding that those most interested in 
participating in these sorting processes are 
the research-oriented faculty. She also 
observed that, “hallway discussions are 
becoming livelier and focus less on ‘What 
content did they know?’ and more on ‘What is 
the whole breadth of criteria for [what] our 
students know and do in our courses?’” 



2 This examination was developed in the 1970s and 1980s 
by Jonathan Warren and was sponsored by the 
Educational Testing Service. 
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Addressing Risks to Faculty 
innovators 

The panelists also explored the risks to 
faculty of shifting to new learning goals, 
teaching strategies (including assessment 
strategies), and student outcomes. This topic 
was introduced by Brock Spencer and 
sustained by several pointed comments and 
questions from the audience. 

In response to expressions of concern 
about the risks of innovation to faculty, 
Mazur described the early stage of 
implementation as posing the greatest 
problems. In his experience, initial efforts at 
innovation often result in lower student 
grades, which, in turn, may translate into a 
lower ranking on traditional faculty 
evaluation forms and into departmental 
pressure to revert to traditional methods. 
Thus, faculty reformers face risks created by 
peer and student disapproval and by peer 
discouragement when improvements are not 
immediate. He encouraged faculty to “stick 
with it,” citing the analogy of a tennis coach’s 
efforts to change an athlete’s grip: at first, 
the athlete is awkward and frustrated, but 
eventually achieves a higher level of 
performance than would have been possible 
with the inferior grip she had perfected 
before the coaching. “Good students” are like 
the tennis player: while facile when using an 
inferior method, they will never achieve 
higher levels unless they rise to the challenge 
posed by methods that require deeper 
conceptual understanding. Like the tennis 
player, faculty too will never achieve higher 
levels in their teaching unless they stick with 
it. 

Audience comments revealed that many 
faculty involved in changing their individual 
courses believe they may be risking or 
wasting their time unless these courses are 
part of a sustained or systemic change. The 
panelists agreed that it is risky for faculty to 
invest their time in isolated efforts and 
offered strategies for overcoming this risk. As 
Mazur put it, “We’re in a situation where we 



have to move a lot of pieces of the puzzle at 
the same time.” All three emphasized that 
sustained and coordinated- change will 
prevail only if a critical mass of faculty is 
involved. 

Ebert-May encouraged Forum 
participants to enlist a critical mass of 
reform-minded faculty through a collegial, 
invitational approach. Invoking a virus 
metaphor, she explained that faculty catch 
the “active learning bug” when they 
experience for themselves the pleasure of 
really engaging their students in learning. 
She cautioned participants against 
inadvertently poisoning the well by using a 
competitive approach, that is, by trying to 
convince colleagues to change by comparing 
their own (implicitly superior) methods with 
traditional methods. This strategy can be as 
destructive for faculty as curve grading is for 
students. 

Mazur observed that it will be difficult to 
develop a critical mass of faculty unless those 
who become engaged in reform are able to 
stay engaged. Faculty will stick with it only if 
they 

• receive support from administrators and 
funders in recognition of their extra work, 

• overcome the lone-reformer syndrome by 
seeking out like-minded colleagues 
elsewhere and work with a regional or 
national network to develop greater 
visibility and acceptance for the new 
methods, and 

• obtain agreement from their colleagues on 
“what it is we want our students to learn, 
how we measure that, and what different 
measurement instruments that exist now 
really tell us.” 

He emphasized that it is destructive to 
blindly assess new goals with instruments 
designed to assess old goals. 

Need for Systemic Alignment 

The panelists concluded their session by 
zooming out from their own course-based 
reform efforts and positioning them within a 



broad systemic context. They initiated this 
shift in scope and perspective by noting that 
development of a critical mass of pro-reform 
faculty is necessary, but not sufficient for 
wide-scale success. Success also depends on 
many other factors, in particular, alignment 
between higher education and K-12 SMET 
reform and public understanding of the 
importance of SMET literacy. With respect to 
connecting with the K-12 system, Ebert-May 
emphasized the important role SMET faculty 
play in preparing the next generation of 
teachers for the K-12 system: “What kinds of 
models we are for the next generation of 
teachers is a piece of the complex puzzle Eric 
mentioned.” With respect to the need for 
public understanding of the importance of 
SMET literacy, Mazur showed a video clip 
indicating that (at least in some quarters) 



such an understanding is lacking. The clip 
shows Senate Majority Leader Trent Lott 
forcefully stating to a group of 400 high 
school students that taking four years of 
science and four years of math in high school 
was “a waste of my time, a waste of the 
teachers’ time, and a waste of space.” 
Reminding the audience of a point made in 
Porter’s Forum paper — that failure to 
explore, develop, and adopt new and more 
effective pedagogies poses a great risk to 
individuals, institutions, and society, Mazur 
stated, “Unless SMET faculty and K-12 
teachers change their teaching approaches, 
we will continue to produce students — and 
citizens — who have little appreciation for the 
importance of science, mathematics, 
engineering, and technology.” His 
observation closed this panel session. 



Commentary 

Norman L. Fortenberry 

Director, Division of Undergraduate Education 
NSF Directorate for Education and Human Resources 



We at the National Science Foundation 
are fully cognizant of the expertise that 
resides in this room and across the nation in 
your institutions. After reviewing the papers 
that were submitted and listening to the 
comments raised, my question is, “What can 
be done from an NSF perspective to support 
the types of innovations you have been 
describing?” There have been very exciting 
papers and interesting presentations and lots 
of exchange and interaction. I want to build 
on several of the spoken and written 
comments. 

Diane Ebert-May spoke of a need for 
higher education to join the reform of 
assessment. Concurring with the remarks 
made by Luther Williams, I would say we 
should more broadly conduct educational 
research to encompass higher education. 

Most of the educational research currently 
conducted is focused on K-12. In addition to 
the well-established community of 
educational researchers, NSF's EHR 



Directorate supports a growing number of 
science, mathematics, and engineering 
faculty conducting educational research 
within the context of their disciplines. The 
people at this table are evidence of this trend. 
Research on Educational Policy and Practice 
is a program administered within the 
Division of Research, Evaluation and 
Communication (REC) that affects other 
EHR divisions as well. This program has the 
potential to link undergraduate faculty with 
the results of educational research at the 
undergraduate level. 

David Porter made the points about the 
need for research to inform practice and 
about our need to predicate questions of 
student learning on changes in the individual 
student. A review of EHR-supported research 
projects by NSF Director Neal Lane earlier 
this year illustrates NSF’s attempts to 
address this need. EHR showcased three 
types of research: traditional research within 
SMET disciplines, research on human 
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learning (especially in SMET disciplines), 
and research on applications of technology. 
Among the projects discussed by EHR were 
several supported by the DUE that feature 
teaching and learning within SMET 
disciplines — for example, the work in physics 
by Eric Mazur and Lillian McDermott. We’re 
seeing synergy among the disciplines in 
diffusing educational innovations, not at the 
level of richness that we would like to see, 
but it is a beginning. Other projects indicated 
that support for traditional research within 
SMET disciplines is informing curriculum 
development — for example, Art Ellis’s work 
with Materials Chemistry. All of these efforts 
support the broader goal of integrating 
research and education. DUE’s Course, 
Curriculum, and Laboratory Development 
program will support projects that integrate 
research (including educational research) 
with curriculum development and 
implementation. 

Eric Mazur talked about using a 
publication review model for evaluating 
student work. This model can be related to 
parallel examination of how faculty can be 
evaluated for their full range of 
responsibilities. For example, at Oregon 
State University, rather than simply relying 
on publications, faculty have redesigned their 
tenure and promotion guidelines to evaluate 
peer-recognized intellectual work 
appropriately disseminated. This very broad 
criterion can stand up to tests of rigor and 
allows appropriate evaluation not only of 
traditional discipline-based research, but also 
of pedagogic work; it is applied to the work of 
administrators as well as the work of faculty. 
The criterion came about through 
appropriate use of terms and recognizing 
what the academic community really 
values not publications as publications, but 
intellectual work recognized by peers, 
appropriately disseminated. 

DUE and REC, both in the EHR 
Directorate, are planning a very small 
conference to bring together traditional 
educational researchers and those in the 
disciplines to bridge our knowledge base on 



the how and why of best practices. Such 
bridge efforts will assist faculty in achieving 
Porter’s goal of creating rich syllabi that offer 
students options and opportunities to master 
the material and to acquire individual and 
group skills. 

Ebert-May spoke about students’ need to 
reflect on their own understandings and 
abilities in providing feedback to faculty. She 
also made an interesting case for treating all 
students as prospective K— 12 teachers; in a 
teacher preparation curriculum, you not only 
teach content to students, but you teach them 
to reflect on that content and how they would 
transmit that knowledge. Most of us agree 
that you really learn something when you 
teach it, so we encourage all of our students 
to think about how they would teach what we 
are teaching them. NSF is encouraging that 
inculcation of information, treating all of our 
students as potential teachers. 

But Eric Mazur warned us that great 
change often results in upheaval and 
rejection. It is hoped that DUE programs are 
assisting individuals to surmount these 
challenges. In particular DUE is supporting 
adaptation and implementation of proven 
reforms to new local contexts via our new 
Course, Curriculum, and Laboratory 
Improvement program and by supporting 
more localized efforts within the NSF 
Collaboratives for Excellence in Teacher 
Preparation. We also hope that the 
restructured programs will facilitate more 
coherent approaches to projects that are 
consistent with addressing all the elements 
required to achieve high quality teaching and 
learning outcomes. 

Williams indicated a collaboration 
between the EHR and the CISE Directorates. 
DUE recently committed $500,000 to 
working with the multiagency Digital 
Libraries Initiative to explore educational 
applications of digital libraries. The Test 
Beds in Education activity is focused on 
exploring the feasibility of developing a 
national resource that supports curricular 
and instructional innovation, validation, 
dissemination, and adaptation. It should be a 
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wonderful platform for conducting research 
on teaching and learning. 

Porter noted that every system is 
designed to yield the results observed; in 
DUE, our effort is to be more conscious about 
the systems that we design in support of 
achieving enhanced SMET learning by all 
students. We are devoting increased 
attention to the means we use to measure 
outcomes. We are developing more rigorous 
quantitative and qualitative metrics. Elaine 
Seymour indicated that there had not been as 
much evaluation as we would like in earlier 
NSF efforts. Across our programs, DUE is 
requiring rigorous evaluation. We hope to 
soon take the next step and provide metrics 
up front that principal investigators (Pis) 
will be encouraged to use. Available metrics 
should not only strengthen the evaluation 
done by Pis from a project-based formative 
and summative perspective, but also allow 
greater input to an NSF-focused program 
evaluation across individual projects. One 
example of “dual use” data is an online 
Evaluation Resource Library being developed 



by REC that we hope will be of great use to 
Pis. 

But perhaps more important than 
measuring outcomes is being more 
intentional about the inputs, the educational 
experiments, that we support. As always, we 
will continue to provide strong support for 
individual investigators and novel ideas from 
the field. Such an approach is a defining 
characteristic of NSF programs. However, 
congressional requirements for greater 
accountability and assessment of alignment 
of programmatic goals with programmatic 
(not individual project) outcomes requires 
that we devote heightened attention to where 
those ideas fit within the broader portfolio we 
are supporting. Thus, the questions 
associated with the NSF review criteria 
remain dominant. However, program officers 
must also consider how the projects we 
support inform our organizational 
effort to provide a rich knowledge base of 
effective teaching and learning strategies to 
the SMET community. 
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Panel 2: Assessment and the Promotion of Change 
in Departments, Disciplines, and Institutions 3 



Panel Discussion Summary 

Susan B. Millar 



Panel Facilitator: Elaine Seymour, NISE College Level One, Director, 

Ethnography & Evaluation Research, University of 

Colorado-Boulder 

Panel Members: Brian P. Coppola, Associate Professor of Chemistry, 

University of Michigan 

Eileen L. Lewis, Professor of Chemistry, Canada College 
Richard Tapia, Noah Harding Professor of 

Computational and Applied Mathematics, Rice 
University 



Abstract 

The members of Panel Two focused on 
strategies that they see having the greatest 
significance for improvement in SMET higher 
education. They offered three strategies that 
are useful at all levels and others that are 
specifically suited to individuals, 
departments, institutions, or whole systems. 
The three change strategies that apply at all 
levels are using assessment data as a driver 
and key resource for educational 
improvement; evaluating and improving 
current evaluation criteria; and having the 
will to make change as well as knowledge 
about how to make change. The panelists 
advocated three improvement strategies for 
use by individual faculty and faculty groups: 
focus on efforts that help students through 
major academic transition periods; get 
started using any possible path for course 
improvement; and have a clear destination in 
sight at the outset. While acknowledging that 
change strategies used by individuals are 
important, the panelists also stressed that 
such strategies are unlikely to prevail unless 



supported by change at higher organizational 
levels. To effect change at higher levels, they 
argued that changes in departmental values 
and reward structures are the most critical. 
They also proposed that departments hire 
faculty trained in discipline-centered 
pedagogy to change how we foster student 
learning and advocated that four-year 
institutions work with two-year institutions to 
engender improvements in the quality of 
higher education for the large number of' 
students served by two-year institutions. Just 
as they emphasized that change strategies 
used by individuals are insufficient unless 
supported by change at department and 
institutional levels, the panelists explained 
that department- and institution-level change 
can only be effective as part of a systemic 
effort that supports it. All three panelists 
argued that higher education is morally 
obligated to play its part in a society wide 
effort to solve the serious national problem 
posed by science illiteracy. The solution 
entails a shift to teaching strategies that 
present science as relevant and exciting — as 
opposed to distant and dull. Higher education 



3 The papers written by the three members of this panel appear in Appendix A. 
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leaders at all organizational levels also must 
work to align their different value systems to 
develop and enact effective and coordinated 
strategies. 

Seymour set the stage by explaining that 
this panel would consider the effect of 
departmental and institutional values on 
efforts to improve undergraduate SMET 
education and on theories of change in higher 
education . 4 She then asked Forum 
participants to describe the things that their 
departments value. All but three answers 
from the audience indicated a departmental 
focus on research and faculty autonomy. Of 
the “minority” voices, one cited increased 
enrollment as an important departmental 
value, and two described their departments 
as valuing the needs of students and society. 
Seymour next asked what factors Forum 
participants viewed as facilitators of and 
barriers to change. Members of the audience 
included among facilitating factors the high 
value faculty place on student learning, 
leadership at faculty and administrative 
levels, regular conversation and collaboration 
between SMET faculty and faculty in other 
disciplines, pressure from students who have 
had more exciting courses, and external 
money for faculty development. Barriers 
included the reward system, faculty 
hierarchy, and lack of money to support the 
time faculty need to revise curriculum and 
develop new pedagogical and student 
learning assessment techniques. 

With this audience prethinking exercise 
as their launch point, and guided by 



4 More explicitly, the panel had been charged to address 
the following three issues: (1) Learning assessment and 
reform: What role can the assessment of learning play in 
undergraduate education reform — in departments, among 
colleagues, and within the disciplines? (2) Assessment as 
argument: How do we best leverage change in these 
spheres? Can assessment data be used to convince 
colleagues, protect innovators from risk, and build support 
for educational change? and (3) Effective dissemination: 
What forms of dissemination work best in encouraging 
others to try, to support, or (at minimum) not to obstruct 
classroom innovation? 



occasional questions from Seymour, the 
panelists articulated strategies for promoting 
change in postsecondary SMET education, 
occasionally also stating the change theories 
on which these strategies are based. Most of 
the strategies that they advocated were 
intended for use at particular organizational 
levels, individual to systemwide, and a few 
were intended for use at all levels. This 
summary begins with the latter and proceeds 
to the former. 

Change Strategies that Apply at All 
Levels 

1. Assessment data should drive reform. 
The three panelists asserted that at all 
organizational levels, from the individual to 
the national, assessment data are an 
essential resource for and driver of 
educational improvement. Both Lewis and 
Tapia saw current efforts to improve college- 
level SMET education as hampered by the 
lack of basic assessment data. Faculty need 
assessment methods that can ascertain what 
students do and do not know, where their 
problems lie, and how best they can be 
helped. They also need assessment data to 
demonstrate that reform is making a positive 
difference to student learning. 

According to Tapia, the NSF also believes 
that collection and use of evaluation data are 
essential factors in successfully 
institutionalizing improvements. It seeks to 
act on this belief through the design and 
implementation of its grant programs. He 
also observed that NSF has not yet been 
successful in enforcing policies requiring Pis 
to make effective use of evaluation. A critical 
problem in implementing these policies is 
that SMET faculty do not yet know how to 
conduct program evaluation and need help in 
developing student learning assessments 
appropriate to their course learning 
objectives. Both Lewis and Tapia emphasized 
the importance of using third-party 
evaluation experts when seeking to provide 
evidence of the efficacy of education reforms, 
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because outside evaluators provide more 
objective data. 

Daryl Chubin, the panel discussant, 
responded to Tapia’s remarks on behalf of the 
NSF. The NSF, he explained, encourages the 
use of external evaluators for education 
reform projects to help grant recipients think 
systematically about what they are learning 
and to give faculty, who are so close to the 
projects, someone else to help them develop 
the lessons. Chubin added, however, that 
evaluators are not likely to be effective at 
dissemination and that Pis should enlist the 
help of communication specialists who can 
devise strategies needed to communicate the 
lessons learned to diverse audiences, 
including the department, campus, region, 
and academic discipline associations. 

While strongly endorsing the critical role 
of assessment data in higher education 
reform, Coppola cautioned that widespread 
appreciation of the role that high quality 
analyses of assessment data can play in 
improving education will require cultural 
changes in the educational establishment. 
Noting that people do not assign value to 
things they do not understand, he observed 
that changing this feature of higher 
education culture will require a critical mass 
of faculty and administrators who (a) have 
access to evaluation studies that analyze the 
interactions among the many critical factors 
involved in making change and (b) develop an 
appreciation of the kinds of evidence that 
assessment and evaluation practices can 
offer. 

2. Evaluating current evaluation criteria. 
Tapia focused on a second strategy that he 
believes is critical and applicable to all 
organizational levels: evaluating the nature 
and utility of prevailing evaluation 
instruments. He advocated approaching 
change “by evaluating whether the commonly 
accepted evaluation criteria foster or hinder 
the outcomes we value.” While arguing that 
this strategy should be applied at all levels, 
he used an institution-level example to 
illustrate the importance of this approach. 

He criticized the practice common at 



exclusive universities of relying heavily on 
SAT scores, which are a one-dimensional 
evaluation tool. He explained that a Rice 
University study found that, although this 
admissions instrument can predict that 
students scoring below 850 are unlikely 
to do well, it cannot predict the likelihood of 
success for students with scores above a 
threshold of 1050. On the basis of this study, 
Rice University developed strategies to 
compensate for the shortcomings of the 
standardized admissions exam. First, SAT 
scores are used only as a “threshold 
indicator,” and other indicators are used to 
select from among students who pass that 
threshold. Second, all new students are 
required to take diagnostic exams. These 
exams help Rice to place talented students 
who lack certain skills or knowledge into 
courses that will give them the best chance to 
fulfill their potential. Tapia reported that 
this combination of strategies is working: 
most Rice students graduate and go on to 
successful careers in industry and 
government as well as academia. 

3. The will to action. Coppola noted a 
third change strategy that applies to all 
organizational levels: knowledge of what 
works must be combined with the will to 
change. He likened institutions of higher 
education to physicians who smoke, do not 
exercise, and are overweight. They have all 
the knowledge that they need to establish 
and maintain health, but lack the will. His 
point was that “both skill and will are 
needed.” 

Strategies Effective for Individual 
Faculty and Faculty Groups 

The panelists advocated three grass-roots 
strategies in which faculty can engage on 
their own or in small groups. 

1. Focus on student transitions. Tapia 
promoted a strategy of change that places 
students at the center of attention. In 
pursuing this strategy, he has found that 
faculty can maximize their positive impact on 
students — particularly minority students — by 
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selectively focusing their efforts on students 
who are in transition. By arranging for 
support groups and providing mentoring 
during the six- to nine-month high 
school/college and college/graduate school 
transition points, faculty can help these 
students “go perfectly on their way.” 

2. Just get started. Lewis observed that 
there is no one way to go about reform and 
advocated a just-get-started approach to 
change. The important thing is “to take those 
first few steps,” regardless of one s level of 
knowledge, commitment, or understanding of 
where the initial steps will lead. Speaking as 
someone who has gone down this path, she 
explained that 

you become committed to this journey in one 
way or another. Once you have been in a 
classroom where the students are t alkin g 
about ideas, where there is dynamic 
interaction, where you see the excitement on 
their faces, you can’t go back to a classroom 
where they are all sitting there with the 
glazed look, writing notes that, as someone 
said this morning, go straight from the 
instructor’s lecture notes to the students 
lecture notes with no cognitive processes in 
between. 

Once engaged in this process, Lewis 
observed that faculty will go on to seek out 
and even create the resources they need to 
further develop their new teaching strategies 
and spread them to colleagues. They seek out 
like-minded colleagues, investigate the 
emerging literature on postsecondary 
teaching and learning, team up with science 
education colleagues who have expertise in 
teaching and learning and evaluation, attend 
and give workshops, and gather and use 
evaluation data. By drawing on these 
resources, SMET faculty can avoid going 
down blind alleys and can improve their 
effectiveness. 

Lewis theorized that the people most 
likely to start moving down a path that 
engages them in more interactive approaches 
to learning are those who, for one reason or 
other, are on the; sidelines or have a different 



agenda. According to her “fringe theory of 
change,” such instructors tend to listen more 
to students . . . and really pay attention.” As 
a result of this different type of interaction 
with students, they “begin to get a different 
view of learning” and begin to do small 
things differently.” 

3. Have a clear goal. While agreeing with 
Lewis that many different promising higher 
education change strategies are evolving, and 
that any one of them might work well for a 
given person, Coppola disagreed with her 
just-get-started approach. He believes that it 
is critical to start with a clear destination in 
sight. He concurred with the point made in 
Panel One that alignment of goals, 
instructional strategies, and assessment 
practices is critical to successful education 
reform, but offered one caveat: the greatest 
priority should be placed on articulating the 
goals. Stressing once more the value of 
assessment data, he observed that, while 
these data “can tell you exactly where you 
are, [they] will do you no good unless you 
have a clear notion of destination. 

Departmental and Institutional 
Strategies 

While emphasizing that change strategies 
used by individual and ad hoc faculty groups 
are important, the panelists also stressed 
that grass roots reform efforts ultimately will 
amount to little unless they are supported by 
change at higher organizational levels. 

1. Changing the departmental reward 
structure. All three panelists strongly agreed 
with Tapia’s assertion that rewards both 
“drive the whole system” and “are 
determined at the department level.” Tapia 
supported these statements by noting that 
college presidents do not affect faculty 
behavior by exhorting faculty to foster 
student learning; it is the departments that 
determine faculty behavior by making the 
tenure and funding decisions, and 
departments are not held responsible for 
realizing campus vision statements. Funding 
organizations such as the NSF and AAAS 
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also affect faculty behavior because “funding 
confers credibility.” The bottom line, 
therefore, is that departmental rewards are 
critical in effecting a sustained change in 
teaching practices. He reluctantly advised 
junior faculty not to endanger tenure through 
involvement in education reform early in 
their faculty careers because those who are 
not there can’t help change the system. Tapia 
reiterated the need to change the 
departmental reward system to meet the 
needs of all students. He argued that 
achievement of this goal will require the 
commitment of one or two members in every 
department, and that this level of 
commitment will not be achieved without a 
change in the reward system. 

2. Recognizing the contradictory effects of 
funding. Coppola elaborated on the impact of 
funding practices on education change. He 
observed that, when funds for a change 
program are provided by an external source 
and the department fails to assume the 
program cost in the end, external funding can 
act as a double-edged sword: even if 
evaluation data demonstrate that the 
program is effective, it loses credibility when 
it loses funding. Coppola observed that, in a 
grants-driven “funding confers credibility” 
culture, the take-home message is that “what 
is credible is getting on to the next funded 
thing.” 

3. Recruiting discipline-centered 
specialists. Coppola observed that 
departments can use faculty recruitment 
decisions to foster change. He hypothesized 
that, just as departments continuously shift 
the direction of their research interests by 
hiring faculty trained in collateral 
disciplines, institutional change in “how we 
foster student learning will come by treating 
discipline-centered pedagogy as an emergent 
area within the disciplines and hiring faculty 
with expertise in that area.” 

4. Collaborating across two-year and four- 
year institutions. Shifting the focus from 
cross- or multidisciplinary interactions 
within departments to interactions across 
types of institutions, Lewis argued that 
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collaborations between four-year and two- 
year institutions can act as a powerful lever 
for change in educational practice. She 
believes such cross-sector interactions are 
critical because half of all U.S. college 
graduates begin their college careers in two- 
year institutions, and because these 
institutions, although teaching-focused, are 
inclined to adopt the pedagogy and 
curriculum used in four-year institutions. 

She observed that, however promising it may 
be, this cross-sector change strategy has been 
seriously hampered by a history of weak 
communication between four-year 
institutions and community colleges about 
reform and assessment goals. 

Systemic Strategies 

Consistent with their view that the 
change strategies of individuals and ad hoc 
faculty groups will founder unless supported 
by change at department and institutional 
levels, the panelists argued that change at 
these levels will be effective only if it is part 
of an encompassing systemic effort. 

1. The necessity of system change. Coppola 
argued that even if many individuals, 
departments, and even institutions work 
continuously to remediate students and 
faculty who have been “damaged by a flawed 
system,” the underlying problems will 
persist. He believes that we will achieve 
science and mathematics literacy for all only 
by changing the system in which the 
students and faculty are trained. 

2. The offering of science for all. Seymour 
asked whether the generally elitist and 
meritocratic traditions of higher education 
are in conflict with reform efforts motivated 
in part by the goal of enhancing science 
literacy. The panelists responded that, 
although a conflict may be implied by current 
higher education practices, the mission of 
higher education certainly is not in conflict 
with this goal. On the contrary, higher 
education will fail to achieve its mission 
unless it plays its part in a societywide effort 
to achieve this goal. Coppola argued that 
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